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Abstract —  The  presented  elTorl  U  iimed  at  establishing  a 
framework  in  order  to  restore  underwater  imagery  to  the  best 
possible  level,  working  with  both  simulated  and  Iteld  measured 
data.  Under  this  framework,  the  traditional  image  restoration 
approach  is  extended  by  incorporating  underwater  optical 
properties  into  the  system  response  function,  specifically  the 
point  spread  function  (PSF)  in  spatial  domain  and  modulation 
transfer  function  (MTF)  in  frequency  domain.  Due  to  the 
intensity  variations  involved  in  underwater  sensing,  denoisang  is 
carefully  carried  out  by  wavelet  decompositions.  This  is 
necessary  to  explore  different  effects  of  restoration  constrains, 
and  especially  their  response  to  underwater  environment  where 
the  effects  of  scattering  can  he  easily  treated  as  either  signal  or 
noise.  The  images  are  then  restored  using  measured  or  modeled 
PSFs.  An  objective  image  quality  metric,  tuned  with 
environmental  optical  properties,  is  designed  to  gauge  the 
effectiveness  of  the  restoration,  and  serves  to  check  the 
optimization  approach.  This  metric  utilizes  previous  wavelet 
decompositions  to  constrain  the  sharpness  metric  based  on 
grayscale  slopes  at  the  edge,  weighted  by  the  ratio  of  the  power  of 
high  frequency  components  of  the  image  to  the  total  power  of  the 
image.  Modeled  PSFs,  based  on  Wells'  small  angle 
approximations,  are  compared  to  those  derived  from  Monte 
Carlo  simulation  using  measured  scattering  properties.  Initial 
results  are  presented,  including  estimation  of  water  optical 
properties  from  the  imagery-derived  MTFs,  and  optimization 
outputs  applying  automated  restoration  framework 

Keyn^rds-  ocean  optics;  scattering;  image  restoration; 
m&dMlaiion  transfer  function;  point  spread fttnahn;  NfRDD 

L  Introduction 

Due  to  environmental  conditions  arising  from  different 
water  types  and  associated  in -water  optical  properties,  the 
ability  to  generally  extend  the  performance  range  as  well  as 
retrieve  environmental  information  from  underwater  clcctro- 
optical  system  is  difficult.  This  capability  however  is 
important  for  many  civilian  and  miltlary  applications, 
including  target  detection  (e.g.  mine  delation),  search  and 
rescue,  and  diver  visibililyli].  Although  iradilional  image 
enhancement  techniques  can  still  be  used  for  imagery  obtained 
from  underwater  environments,  without  knowledge  of  any 
processes  involved  or  the  optical  properties,  the  effectiveness  is 
considerably  restrained.  The  main  challenge  working  with 
underwater  imagery  results  from  the  rapid  decay  of  signals  due 
to  absorption,  which  leads  to  poor  signal  to  noise  returns,  and 
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the  blurring  caused  by  strong  scattering  due  to  water  and  its 
constituents  which  includes  various  si^cd  particles.  To  properly 
address  this  issue,  knowltxlge  of  in-water  optical  properties  and 
their  relationship  to  the  image  formation  can  be  exploited  in 
order  to  restore  the  imagery  to  the  best  possible  level.  This  in 
turn  provides  much  needed  environmental  information  via 
th rough -thc-sensor  techniques  and  greatly  enhance  current 
operational  capabilities. 

II.  Framework  Components 
A.  Image  Restoration 

Generally  speaking,  a  2-dimenuonal  image  of  an  object  is 
basically  the  combination  of  original  signal,  f(x,yf  convolved 
by  the  imaging  system  response  of  a  point  source,  the  point 
spread  function  or  PSF  hfx,y),  integrated  over  sensor  space  H: 

y,)clx,dy^  ,(i) 

The  system  response  includes  returns  from  both  the 
imaging  system  itself,  as  well  as  the  effects  of  the  medium. 

Mathematically,  it  is  easier  to  manipulate  the  above 
relationship  in  the  frequency  domain  as  the  convolution 
operator  b^omes  a  simple  multiplication.  Applying  a  Fourier 
transform,  the  above  relationship  becomes 

Giu,v)  =  F(u,v)Hiu,v),  (2) 

where  m,  v  are  spatial  frequencies  and  C,  /%  H  arc  Fourier 
transforms  of  g,/and  h  respectively.  The  Fourier  transfer  of 
A,  for  example,  takes  on  the  following  form: 

//(«,v)=J  ~lh{x,y)e-^^^^'“*'^’'cixdy,  (3) 

The  system  response  fiinction  //,  also  referred  to  as  the 
optical  transfer  function  (OTF),  is  the  Fourier  transform  of  ihe 
PSF.  The  magnitude  of  the  OTF  is  the  modulation  transfer 
function  (MTF).  The  MTF  describes  the  contrast  response  of  a 
system  at  different  spatial  frequencies,  and  when  the  phase 
information  is  of  little  concern  as  is  the  case  for  typical 
incoherent  systems,  it  is  a  sufficient  measure  of  the  power 
transfer  Notice  that  the  above  MTF  term  H(u,y}  is  the  total 
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system  response.  Therefore  if  one  views  the  complete  path 
from  target  to  the  bottom  of  eyes  or  the  recording  CCD  plane, 
the  MTF  is  the  effect  of  multiple  individual  components. 
Because  of  the  cascading  nature  of  the  MTF,  in  the  frequency 
domain,  it  can  be  expressed  by  the  direct  product  of  each 
component,  for  instance,  the  optical  system  itself,  and  the 
medium  (plus  any  other  factors  when  applicable): 

//(M,V)=W  (4) 

The  above  formulation,  which  emphasises  the  validity  of 
the  separation  of  the  system  and  the  medium,  is  important  in 
our  analysis.  Usually  the  system  response  can  be 

prc-dctcrmincd  and  calibrated  to  remove  any  si^ifrcani  errors, 
and  in  most  cases,  does  not  vary  with  imaging  conditions. 
Furthermore,  one  should  pay  special  attention  to  the  band- 
limiting  characteristics  imposed  by  such  as  a  camera 

system’s  ficld-of-view,  and  Nyquist  sampling  frequency  limits 
imposed  by  the  CCD  resolution  [2].  From  (2),  one  can  sec  with 
the  knowledge  of  system  MTF  H(^M  and  transformed  image 
output  the  original  image  can  be  theoretically  restored 

by  deconvolving  the  effect  in  frequency  domain  to  obtain  the 
unbluTTcd  version  after  inverse  transform. 

Needless  to  say,  the  presence  of  various  noises  (such  as 
scattering  or  surface  fluctuations)  complicates  these  through- 
the-sensor  techniques.  They  introduce  an  extra  term  in  both  (I) 
and  (2).  The  medium  effect  is  two  fold:  scattering  would 
contribute  extra  blurring  on  top  of  system  response,  while 
attenuation  results  in  reduced  signabto-noisc  ratio,  Ehffcrent 
image  restoration  approaches  exist  to  reduce  and  compensate 
for  the  noise  to  deblur  images,  such  as  Wiener,  Lucy- 
Richardson  and  blind  decoovolutions[2].  Under  our 
framework,  these  approaches  arc  implemented  and  exploited  to 
determine  the  best  approach  working  with  underwater  images. 
In  addition  a  denoising  technique  based  on  the  wavelet 
decomposition  is  applicd[3], 

B.  Modeling  of  System  Response  of  Undenmter 

Environments 

For  circular  symmetrical  response  systems,  such  as  the 
isotropic  volume  scattering  type  found  in  the  seawater,  the 
corresponding  2-dimcnsiOfial  transforms  found  in  (3)  can  be 
reduced  to  a  one-dimensional  Hankel  (Fourier-B(^sel) 
integral, 

//((/, r)  =  2k  "]j^{iKeiff)h{e,r)ede .  (5) 

^=0 

Wells  [4]  applied  small  angle  approximations  to  the  above 
and  derived  a  robust  underwater  modulation  transfer  model 
which  is  briefly  outlined  below.  By  separating  the  exponential 
decay  effect  with  distance  due  to  the  medium,  the  MTF  of  the 
medium  in  (4)  can  be  expressed  as 

=  (6) 

where  D{y/)  is  the  decay  transfer  function  (DTF)  and  is 
independent  of  the  range  of  detection.  This  provides  a  method 
to  compare  measurements  at  different  ranges  for  consistency. 


By  using  a  thin  slab  model  with  the  small  angle  scattering 
approximation,  and  assuming  a  simple  phase  function, 

hd. 

"  2K{dl 

Wells  [4]  showed  that  the  DTF  of  the  seawater  can  be 
expressed  as 


c  - 


(8) 


Of}  is  related  to  the  mean  square  angle  (MSA),  h  is  the  total 
scattering  coefficient,  and  c  is  the  total  attenuation  coefficient 

[5] .  It  has  been  shown  that  the  exact  shape  of  the  scattering 
phase  fimetion  docs  not  affect  the  derived  results  [6].  With  the 
imaging  range  defined,  the  medium  MTF  can  be  obtained  from 

(6) . 


C.  Image  Quality  Metric  flQM) 

To  determine  the  quality  of  restored  images,  besides 
subjective  visual  comparison  which  is  prone  to  significant 
variations  from  different  viewers,  an  objective  quality  metric 
is  needed  for  these  scattering-blurted  images.  This  was  critical 
for  the  development  of  an  automated  restoration  scheme,  since 
the  computer  needs  to  “know”  which  direction  to  “go”  and 
when  to  stop,  on  small  improvement  increments. 

Our  approach  is  a  wavelet-decomposed  and  denoised, 
perceptual  metric  constrained  by  a  power  spectrum  ratio.  More 
details  can  be  found  in  [3],  Briefly,  images  are  first 
decomposed  by  a  wavelet  transform  to  remove  random  and 
some  medium  noise.  This  augments  chances  of  true  edge 
detection.  Sharpness  of  each  edge  is  then  determined  by 
regression  to  determine  the  sIojk  angles  between  ^yscale 
values  of  edge  pixels  versus  location.  The  overall  sharpness  of 
the  image  is  the  average  of  measured  grayscale  angles  (GSAs), 
weighted  (WGSA)  by  the  ratio  of  the  power  of  the 
decomposition  details  to  the  total  power  of  the  image.  Adaptive 
detennination  of  edge  widths  is  facilitated  by  values  associated 
with  image  noise  variances.  To  frirther  remove  the  noise 
contamination,  edge  widths  l<^s  than  corresponding  noise 
variances  or  regression  requirements  arc  discarded.  Without 
losing  generality  while  easily  expandable,  only  horizontal  edge 
widths  are  used  in  this  study. 


D.  Framework  Summary 

The  implementation  of  the  framework  is  termed  NRL 
Image  Restoration  via  Denoised  Deconvolution  (NIRDD).  The 
flowchart  in  Fig.  I  shows  the  process  involved  in  the 
automated  restoration  framework.  The  optimization  process  is 
based  on  the  quality  of  restoration  measured  in  WGSAs.  This 
uses  the  Wells’  model  to  derive  the  medium  MTF  and  then  the 
system  PSF  with  knowledge  of  camcra/lens  MTF.  Both 
automated  and  manual  input  (measured  optical  properties)  can 
be  incorporate!  in  this  framework.  This  framework  can  be 
further  applied  towards  real-time  image  enhancement  in  the 
field. 


Figure  I .  Sketch  of  the  automated  res^toration  framework.  Shaded  blocks 
correspond  to  the  storage  of  optimized  data  during  the  process. 

III.  Initial  Results  and  discussions 

Test  image  sets  were  obtained  using  the  Laser  Underwater 
Camera  Imaging  Enhancer  or  LUCIE  from  Defense  Research 
and  Development  Canada  (DRDC),  during  an  April-May  2006 
NATO  trial  experiment  in  Panama  City^  Florida,  The  amount 
of  scattering  and  absoqjtion  were  controlled  by  introducing 
Maalox  and  absorption  dye  respectively.  Although  the  effects 
of  polarizations  are  examined  during  the  experiment,  ail 
images  used  in  this  study  are  unpolarized.  In-waicr  optical 
properties  during  the  experiment  were  measured.  These 
included  the  absorption  and  attenuation  coefficients  (Wellabs 
ac-9),  particle  size  distributions  {Sequoia  Scientific  LISST- 
100),  and  volume  scattering  functions  (multi -spectral  volume 
scattering  meter  or  MVSM),  Using  the  framework  discussed 
above,  image  restoration  is  carried  out  and  medium  optical 
properties  are  estimated. 

The  measured  MTFs  of  lens  and  LUCIE  camera  system  arc 
used  to  model  the  combing  system  MTF  v)  in  (4)), 

which  is  shown  in  Fig.  2^  modeled  by  a  Gaussian  point 
r^ponse  {/?’>0.99  in  all  fits).  It  is  clear  that  the  camera  is  the 
limiting  factor. 


Figure  2.  Overall  camera  system  (lens  plus  camou)  MTF.  Mcasuied  camera 
and  lens  MTF  were  used  in  Guassjan-type  fits. 


Figure  3.  Measured  PSF  via  Momc  Carlo  {MQ  (solid!  compared  to  modeled 
results  (doilcdK  using  data  from  April  28.  2006  afternoon  experiment. 

The  system  response  functions  (PSFs)  of  the  medium  are 
derived  from  measurement  results  of  the  volume  scattering 
functions  and  Monte  Carlo  simulations  [7J.  Modeled  PSFs 
using  (6)  are  compared  to  the  measurements  derived.  A 
comparison  of  the  modeled  PSF  using  (6)  and  the  in-situ 
measured  result  is  shown  in  Fig.  3.  Note  they  are  in  relative 
units.  The  discrepancy  amongst  the  two  PSFs  might  be  the 
result  of  excluding  the  direct  beam  contribution  in  Monte  Carlo 
simulations,  which  inherently  reduces  the  peak  contribution  of 
non -scattered  photons.  The  effect  of  multiple  scattering  which 
is  accounted  for  in  Monte  Carlo  approach  also  helps  to  reduce 
the  PSF  peak.  In  cither  case,  they  are  affi^ted  by  the  sampling 
frequency  limits  imposed  by  detectors  in  spatial  domain. 


Figure  4.  Sample  original  (top),  restored  images  based  on  measured  PSF  and 
modeled  (bonom),  wiib  WGSA  values  ofO.OS  and  0.14  respectively. 


The  images  are  restored  using  PSFs  derived  from  both  ihc 
modeled  and  measured  optical  properties,  and  then  quantified 
by  the  image  quality  metric  discussed  earlier.  A  sample  pair  is 
shown  in  Fig.  4,  with  corresponding  WGSA  values  0,05  and 
0J4  respectively.  The  visual  restoration  differences  between 
measurement  derived  PSFs  and  modeled  PSFs  arc  small 
despite  the  differences  in  PSFs  (Fig  J),  thus  only  one  is  shown. 
Further  details  can  be  found  in  [3], 

An  optimization  approach  is  used  to  estimate  underwater 
optical  properties.  The  forward  scattering  and  the  mean  square 
angles  are  used  for  initial  testing,  A  set  of  individual  images 
obtained  under  different  conditions  or  ranges  is  used.  Via  the 
pathway  shown  in  Table  1,  optimization  on  the  image  metric  is 
carried  out  Table  1  compares  the  retrieved  optical  properties 
with  those  measured  in-situ.  While  the  general  trend  matches 
well,  significant  deviations  do  exist  (eg  0.35  versus  1.0),  and  is 
part  of  ongoing  investigations.  Possible  causes  include 
deficiencies  in  denoising,  the  criteria  used  in  the  deconvolution 
algorithms,  level  of  wavelet  depositions,  and  the  edge  detection 
algorithms.  Automated  batch  processing  of  images  obtained 
within  the  same  lime  frame  should  also  improve  the  retrieval. 


0=0,27  n\  \  at  low  angular  frequ<mcies  (Fig.  5).  Clearly  the 
above  result  can  benefit  from  measurements  at  increased 
spatial  frequencies.  Higher  dynamic  ranges  will  also  help 
eliminate  probable  digitization  errors 


Figure  5.  Sample  rcsull  of  retrieved  optical  propcnics  from  measurod  MTFs 
based  on  WelU"  small  angle  seattcring  iheofy.  Top  and  bottom  curves 
comspond  to  turbid  (c=0.95  m  ')  and  clear  (e^.55)  conditions  respectively. 


TA»Li- 1 .  Estimated  optical  properties  bv  optim  ized  image 

RtSlORATION  VERSUS  MEASUREMENTS 
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ID 

range 

r(m) 

measured 

b(m^} 

cstimaicd 

b(ni'') 

estimated 

MSA 

258.% 

5.5 

0.56 

0.6 

0.01 

73240 

.1.9 

0.95 

0.9 

0.02 

63402 

5.J 

0.95 

0.7 

0.02 

72328 

7.5 

0.35 

LO 

0.01 

In  addition,  it  is  straightforward  to  obtain  medium  optical 
properties  from  the  imagery-nderived  DTF,  by  applying  the  first 
order  Taylor  expansion  to  the  exponent  for  undo:  Wells’ 
formulation  {1 1), 


IV,  Summary 

An  automated  restOTalion  framework  for  underwater 
imagery  is  implemOTted,  along  with  through-thc-sensor  optical 
properties  retrieval.  Issues  special  to  underwater  imaging  such 
as  denoising  and  image  quality  assessment  are  addressed  The 
model  includes  the  responses  of  the  camera  as  well  as  medium. 
Analytical  modeling  results  compare  favorably  to  Monte  Carlo 
simulations  based  on  measured  in-situ  optical  properties.  Lastly 
initial  results  presented  support  the  effectiveness  of  our 
imaging  analysis  framework  even  though  further  improvements 
are  ne^cd  to  improve  restoration  quality  and  accuracy  of 
optical  property  retrievals. 
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Taking  the  following  regression  equation  form  following  (S) 

Z)(.T)  =  C  +  ^fcf^  (10) 

results  are  shown  in  Fig.  5.  The  regression  parameters  for  the 
clearer  water  arc  ^=-33.47  and  00.3989.  For  the  turbid 
setting,  they  arc  i4=-44.l8  and  00.7446.  This  approach  would 
yield  c'=0.40  m'^  for  the  clear  water  situation,  inline  with  the 
measurement  m*').  For  the  turbid  situation,  the 

regression  yields  cHJ.74  m" ,  which  is  smaller  compared  to  the 
measured  value  of  0,95  m  \  For  absorption  under  the  turbid 
condition,  the  regression  trend  is  close  to  the  field  measured 
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